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IMAGE PROCESSING METHOD FOR REMOVING GLASSES 
FROM COLOR FACIAL IMAGES 

TECHNICAL FIELD 

The present invention relates to image processing fields, and more 
particularly, to an image processing method for removing glasses from a color facial 
image by using recursive principal component analysis (PCA) reconstruction. 

BACKGROUND ART 

As an information-oriented society has been come, person identification 
techniques for discriminating a person from others have become more important, and, 
thus there have been significant number of studies in the field of personal 
information protection and person identification through a computer using 
biometrical technologies. In biometrical technologies, facial recognition technique 
becomes the most convenient and competitive technique since it does not require a 
specified action or behavior of a user and employs a non-contact manner. The 
facial recognition technique is widely used in various applications such as 
identification, human-computer interface (HCI), and access control. However, 
there are several drawbacks in the facial recognition technique. One of these 
drawbacks is deformation of facial images occurred by glasses. 

To remove glasses from a facial image with the glasses, various image 
processing methods are proposed: an algorithm for extracting glasses from a facial 
image using a deformable contour to remove the extracted glasses; an algorithm for 
removing small occlusion regions such as certain facial regions occluded by glasses 
using a flexible model that is called as an active appearance model; and an image 
processing method using PCA algorithm. 

An image processing method using PCA algorithm is now widely used. 
The PCA algorithm is classified into two processes. One is a training process for 
extracting eigenfaces from a plurality of unspecified sample glassless facial images 
T N , wherein N=1,2,...,M. The sample facial images T N include facial images of 
an individual and/or another individuals. The other is a process for obtaining 
glassless reconstruction images from current input facial images T with glasses by 
using the extracted eigenfaces. 

Descriptions of the training process for extracting eigenfaces will be first 
described in detail. An average image <p are calculated from the sample facial 
images V N for use in the training process by using Equation 1 and the average 
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image <p is subtracted from the sample facial images T N as expressed in Equation 
2, wherein each of the sample facial images T N is expressed as a column vector. 

i M 

A/ " 

5 

a>„=r„-?> (Eq.2). 

Then, a covariance matrix C with respect to the sample facial images T N is 
obtained from differential images O n , which is calculated by subtracting the 
1 0 average image <p from each of the sample facial images T N by using the 
following Equation 3. 

1 m 

M f£ (Eq. 3) 

A = [0 1 ,0 2 ,.-,0 Af ] 



1 5 wherein A is a matrix composed of the differential images <& N and A T is a 
transpose of A. 

Consequently, eigenvectors are obtained from the covariance matrix C, 
wherein the eigenvectors is referred to eigenfaces u k (k=l,---,M). Detailed 
description for a process of obtaining the eigenfaces u k will be omitted because this 
2 0 process is well known to those skilled in the art. 

Next, the input facial images T with glasses are expressed as glassless 

reconstruction images f by using the eigenfaces u k . With the following 

Equation 4, the average image q> is subtracted from the input facial images T , and 
the resultant is projected to the respective eigenfaces u k . 



25 



afc=nr<r-*0, 

wherein co k is a weight which allows the input facial images Y to be expressed 
a space consisting of the eigenfaces u k . The reconstruction images f are also 
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expressed in terms of the sum of weights of the eigenfaces u k extracted from the 
sample facial images Y N by using the following Equation 5. 




(Eq. 5) 



M'<M 



5 



wherein a number of the eigenfaces u k required is equal to M or to M' less than M , 
M being a total number of the eigenfaces u k . 

Where eigenfaces u k are extracted from the sample facial images T N as 
described above, the extracted eigenfaces u k include facial characteristics only so 
10 that final glassless facial images can be obtained by reconstructing the input facial 
images Y on the basis of the extracted eigenfaces u k to produce the 

reconstruction images r . However, the reconstruction images r produced 
according to the conventional method have many errors thereon. Referring to Fig. 1, 
which shows that glasses are not removed completely although the reconstruction 

1 5 images T are similar to the input facial images T , and there are numerous errors 
over the reconstruction images f . In Fig. 1, "client" is a person included in a 
training set and "non-client" is a person excluded in the training set. Although there 
are numerous errors as shown in Fig. 1, the reconstruction images f of "clients" are 
better than those of "non-clients" in quality since facial characteristics are reflected in 

2 0 the extracted eigenfaces u k . 

However, there are some problems in regarding the reconstruction images 
r obtained according to the conventional method as complete glassless facial 
images. Firstly, if the reconstruction images T are generated with respect to the 
input facial images T on the basis of the eigenfaces u k that are extracted from the 

2 5 sample facial images T N included in the training set, particular characteristics of the 

input facial images T would not be appeared on the reconstruction images T . 
Secondly, if occlusion regions due to glasses are considerable in the input facial 
images T , the reconstruction images T will include many errors thereon so that 
these may appear to be unnatural and different from the input facial images T . 
30 As described above, since problems due to glasses in the input facial images 

r are merely regarded as the matter of glasses frame, many limitations are arisen in 
the conventional methods so that obtaining high quality glassless facial images is 
very difficult. 
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DISCLOSURE OF THE INVENTION 

It is, therefore, an objective of the present invention to provide an image 
processing method using recursive PCA reconstruction, which is capable of obtaining 
glassless color facial images with a high quality resolution similar to input glasses 
color facial images, by removing all occlusion regions including not only glasses 
frame region but also occlusion regions due to reflection by lens and shades by 
glasses within the input glasses color facial images. 

In accordance with the present invention, an image processing method for 
obtaining a glassless image from a color frontal facial image bearing glasses, 
comprising the steps of: a) receiving an RGB color frontal facial image bearing 
glasses, wherein RGB are red-, green-, and blue-component contained in the received 
RGB color frontal facial image; b) extracting candidates of eye regions from the 
received RGB color frontal facial image; c) determining an exact eye region out of 
the candidates and normalizing the received RGB color frontal facial image in a 
predetermined size by centering on the determined eye region; d) extracting a glasses 
frame region by using color information contained in the received RGB color frontal 
facial image and edge information of a glasses frame; e) performing an RGB-HSI 
transformation on the normalized frontal facial image; f) generating H\ S\ and T 
glassless compensated images on the basis of the RGB-HSI transformed H-, S-, and 
I-component normalized frontal facial images, wherein the H-, S-, and I-component 
represent a hue, a saturation, and an intensity, respectively; g) obtaining R', G', and B' 
compensated images by performing an HSI-RGB transformation on the H', S', and T 
glassless compensated images; and h) creating a glassless final color facial image on 
the basis of the R', G\ and B ? compensated images. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The above and other objects and features of the present invention will 
become apparent from the following description of preferred embodiments given in 
conjunction with the accompanying drawings. 

Fig. 1 is input facial images with glasses and glassless reconstruction facial 
images obtained according to a conventional image processing method using PCA 
reconstruction. 

Fig. 2 is a flow diagram of a process of removing glasses from color input 
facial images in accordance with an image processing method, which employs 
recursive PCA reconstruction of the present invention. 



Fig. 3 is a diagram for explaining the recursive PCA reconstruction of 
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processing normalized facial images of an intensity component (I-component) in 
accordance with the present invention. 

Fig. 4 is a diagram of a process of extracting a glasses frame region from 
each of the color input facial images in accordance with the present invention. 
5 Fig. 5 is a criterion for determining an occlusion region in each differential 

image on the basis of a gray-level in accordance with the present invention. 

Figs. 6a to 6c are images classified into skin and non-skin color regions by 
using color information of the color input facial images in accordance with the 
present invention. 

1 0 Fig. 7 is a range of weights for compensating the reconstruction facial 

images in accordance with the present invention. 

Fig. 8 is a flow diagram for processing normalized facial images of 

saturation and hue components (S- and H-components) in the recursive PCA 

reconstruction in accordance with the present invention. 
1 5 Fig. 9 is the I-component normalized images and I-component compensated 

images obtained from the I-component input facial images in accordance with the 

present invention. 

Fig. 10 is the input color facial images and final color images obtained from 
the input facial images in accordance with the present invention. 
2 0 Fig. 1 1 is a block diagram of an image processing system in accordance 

with the present invention. 

Fig. 12 is S-component normalized images and S-component compensated 
images obtained from the S-component normalized images in accordance with the 
present invention. 

2 5 Fig. 13 is H x - vector-component normalized images and H x - vector- 

component compensated images obtained from the H x - vector-component normalized 
images in accordance with the present invention. 

Fig. 14 is Hy-vector-component normalized images and H y - vector- 
component compensated images obtained from the H y - vector-component normalized 

3 0 images in accordance with the present invention. 

BEST MODE FOR CARRYING OUT THE INVENTION 

Now, embodiments in accordance with the image processing method of the 
present invention will be described in detail with reference to Figs. 2 to 14. 
3 5 Referring first to Fig. 11, which is a block diagram of an image processing 

system employing recursive principal component analysis (PCA) reconstruction in 
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accordance with the present invention. As shown, image processing system 1000 in 
accordance with the present invention comprises image input unit 1002, image 
processing unit 1004, image output unit 1006, and face database (DB) 2000. 

Image input unit 1002 receives color input facial images with glasses and 
5 transmits them to image processing unit 1004. Image input unit 1002 is 

implemented with a conventional image input device such as a digital camera. 
Image processing unit 1004 performs an image process in accordance with the 
present invention on the color input facial images received from image input unit 
1002 and generates glassless color facial images. Image processing unit 1004 is 

1 0 implemented with a conventional computing device. Face DB 2000 stores the color 
input facial images and intermediate images of the color input facial images 
generated during the image process performed by image processing unit 1004. Face 
DB 2000 also stores glassless sample facial images, which are used in the recursive 
PC A reconstruction to be described in below. Image output unit 1006 outputs 

15 glassless color facial images, which are generated by image processing unit 1004. 

Image output unit 1006 is implemented with a conventional display device such as a 
monitor. 

Referring to Fig. 2, which is a flow diagram of a process of removing 
glasses from input color facial images in accordance with an image processing 
2 0 method of the present invention. The image processing method employs recursive 
PCA reconstruction in accordance with the present invention. Image processing 
system 1000 first receives color input facial images with glasses from image input 
unit 1002 (step S202). 

Image processing unit 1004 of image process system 1000 obtains binary 

2 5 generalized skin color distribution (GSCD) transform images, which enhance skin 

colors of faces in the color input facial images by using color information contained 
therein (step S204). Also, image processing unit 1004 obtains binary black and 
white color distribution (BWCD) transform images, which enhance black and white 
colors of faces in the color input facial images by using the color information (step 

3 0 S206). Obtaining the binary GSCD and BWCD transform images from the color 

input facial images is performed by a known method in the art. 

In order to find candidates of eye regions required for normalizing the color 
input facial images, image processing unit 1004 removes certain regions having 
colors different from the skin color, such as eye, eyebrow, mouth, or slipping-down 
3 5 hairs, from the binary GSCD transform images by performing a morphology filtering 
(step S208). Image processing unit 1004 extracts the candidates from the color 
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input facial images by using the binary BWCD transform images and the 
morphology filtered GSCD transform images (step S210). Image processing unit 
1004 determines exact eye regions out of the candidates and normalizes the color 
input facial images with a predetermined size by centering on the exact eye regions 
(step S214). The normalized color input facial images are represented as primary 
color components of red, green, and blue in a red-green-blue (RGB) model. 

Image processing unit 1004 generates normalized hue (H), saturation (S), 
and intensity (I) components input facial images of by performing an RGB-HSI 
transformation on the normalized color input facial images (step S216). As is well 
known in the art, RGB model images are easily transformed into HSI model images 
and vice versa. In the present invention, in order to process the color input facial 
images, the RGB-HSI transformation is performed on the normalized color input 
facial images by using Equation 6. 




/ = |(/? + G + fl) 



5 = 1 — 



(tf+G+fl) 
H = cos _1 < 



[min(# + G + fl)] 



I[(/?-G) + (/?-i?)] 



(Eq. 6) 



[(R - Gf +(R- B)(G - B)Y 2 



ifXB>G) 
then 

H=360°-H 



wherein H-component has a value of 0 to 360 degrees, S-component has a value of 0 
to 1, and I-component has a value of 0 to 255. 

Meanwhile, image processing unit 1004 extracts glasses frames using the 
color information of the color input facial images and edge information of the glasses 
frames in the color input facial images (step S212). Detailed description of this 
procedure will be made with reference to Fig. 4. 

Referring to Fig. 4, image processing unit 1004 performs an AND operation 
on the morphology filtered GSCD transform images obtained at step S208 and the 
binary BWCD transform images obtained at step S206, to thereby obtain image 400. 
Image 400 shows black and white regions in the color input facial images and 
includes eye and eyebrow regions. Next, image processing unit 1004 performs an 
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OR operation on image 400 and image 100 that is a GSCD transform image of gray- 
level obtained at step S204, to thereby obtain image 801. Image 801 is an image of 
which eliminates eyes and eyebrows from image 100. In order to accurately 
represent the glasses frames included in the color input facial images, image 
5 processing unit 1004 detects edges of the glasses frame in image 801 through the 
well-known Sobel method to generate image 802. Image processing unit 1004 
performs an OR operation on image 802 and an inversion image of image 801 to 
obtain image 803, and then normalizes image 803 with the same size as the 
normalized color input facial images at step S214, to thereby obtain image 800 

10 (hereinafter, referred to glasses frame image G 800) using location information of the 
glasses frame. Glasses frame image G 800 contains a glasses frame region only. 

Referring back to Fig. 2, image processing unit 1004 obtains compensated 
images of H-, S-, and I-components by applying the recursive PC A reconstruction 
with respect to the normalized color input facial images of H-, S-, and I-components 

15 (step S218). Image processing unit 1004 then performs the HSI-RGB 

transformation on the compensated image of H-, S-, and I-components to obtain final 
glassless color facial images (step S220). 

Referring to Fig. 3, which shows a diagram for explaining the recursive 
PC A reconstruction in which image processing unit 1004 processes the normalized I- 

2 0 component input facial images out of the RGB-HSI transform images at step S218. 
First, image processing unit 1004 reconstructs the normalized I-component input 
facial images according to the conventional PC A algorithm. Hereinafter, the 
normalized I-component input facial images are referred to I-component normalized 
images T(i) 500, wherein i is an index to indicate pixels within I-component 

2 5 normalized images T{i) 500. That is, resultant images reconstructed according to 

Equation 5 becomes images corresponding to 601 of Fig. 3 (hereinafter, referred to I- 

component reconstructed images f (i) 601). Using Equation 7, image processing 
unit 1004 calculates differential images between I-component normalized images 
r(/) 500 and I-component reconstructed images f (/) 601, to obtain images 

3 0 corresponding to 602 of Fig. 3 (hereinafter, referred to I-component differential 

images d(i) 602). 



</(/) = |r(o-f(o| 



-8- 



(Eq. 7). 



PCT/KR03/00927 

'RO/KR31 12 2003 



As shown, glasses frames overlapping with eyebrows are not extracted 
completely in I-component differential images d(i) 602. This results from a 
lower difference in gray-level between I-component normalized images T(i) 500 



5 overlapping with the eyebrows in I-component normalized images r(i) 500 are 
represented as eyebrows having low gray-level in I-component reconstructed images 

f (i) 601. If the glasses frames overlapping with the eyebrows are not removed 

completely, facial features may be degraded so that obtaining reconstructed images 
similar to I-component normalized images r(i) 500 is very difficult. In order to 

10 accurately find regions of the glasses frames overlapping with the eyebrows, i.e., 

occlusion regions, the present invention employs glasses frame image G 800 (shown 
in Fig. 4) extracted at step S212 of Fig. 2. 

Image processing unit 1004 stretches I-component differential images d(i) 
602 after reflecting gray-level information corresponding to the face therein, to 

1 5 thereby create images corresponding to 603 of Fig. 3 (hereinafter, referred to I- 
component differential images D(i) 603) according to Equation 8. That is, I- 
component differential images D(i) 603 are created by taking square roots to the 
product of gray-level intensities with respect to I-component differential images 

d(i) 602 and I-component reconstructed images f (0 601. 



Using I-component differential images D(i) 603 is advantageous as 
follows. Emphasizing a difference of gray- level intensities with respect to the 

2 5 occlusion regions due to the glasses frames is possible and thus the occlusion regions 

are easily removed. This results in obtaining more natural looking glassless images. 
Further, decreasing variation of gray-level intensities with respect to the facial 
features, such as eyes and eyebrows, in I-component normalized images T(i) 500 is 
possible so that I-component normalized images T(i) 500 can be used as these 

3 0 stand. 

In order to include glasses frame image G 800 in I-component differential 
images £>(/) 603, classifying I-component differential images D(i) 603 into 



and I-component reconstructed images f (/) 601 since the glasses frames 



20 




(Eq. 8). 
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occlusion regions and non-occlusion regions is necessary. Error distribution in I- 
component differential images D(i) 603 appears in the occlusion regions due to the 
glasses greater than the non-occlusion regions. With error distribution, the 
occlusion and non-occlusion regions are classified in a gray-level range of 0 to 255 
5 shown in Fig. 5. Thresholds for classifying the regions are determined by Equation 
9. 



10 wherein T L and T H are lower and upper thresholds, respectively; D(j) is error 
values in the non-occlusion regions; and D(k) is error values in the occlusion 
regions. 

Detailed description of calculating the lower and upper thresholds T L and 
T H will be followed. In order to find non-occlusion regions in I-component 

15 normalized images T(j) 500, image processing unit 1004 performs OR operation 
with an inversion image (shown in image (a) of Fig. 6) of the binary GSCD 
transform image, and the binary B WCD transform image (shown in image (b) of Fig. 
6). Once more, image processing unit 1004 normalizes the resultant image with the 
same size of I-component normalized images T(i) 500, centering on locations of 

2 0 eyes, to obtain an image (shown in image (c) of Fig. 6). In image (c) of Fig. 6, dark 
regions are obtained by emphasizing the skin color in I-component normalized 
images T(i) 500 and belong to the non-occlusion regions. Accordingly, the lower 
threshold T L is obtained by computing an average of errors occurred in the non- 
occlusion regions within I-component differential images £>(/) 603. 

2 5 On the other hand, the occlusion regions by the glasses are obtained by 

emphasizing the non-skin colors in I-component normalized images T(i) 500 and 
expressed as white regions in image (c) of Fig. 6. Accordingly, the upper threshold 
T H is obtained by computing an average of errors that are greater than the lower 
threshold T L within I-component differential images D(i) 603. 

3 0 Here, uncertain regions still exist in I-component differential images D(i) 

603. The uncertain regions contain errors greater than the lower threshold T L and 
smaller than the upper threshold T H . The glasses frame regions are not contained 
in the occlusion regions since values of gray-level of the glasses frame regions in I- 



T L = mean(D(j)\ 
where, j e skin region 
T H = mean(D(k)), 



(Eq. 9) 
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component differential images D(i) 603 are likely to include errors less than the 
upper threshold T H . Therefore, in order to include the glasses frame regions in the 
vicinity of eyebrows into the occlusion regions, glasses frame image G 800 extracted 
at step S212 is used. In Equation 10, where values out of errors in I-component 
5 differential images D(i) 603 have a gray-level less than the upper threshold T H , a 
value G(i) having a gray-level greater than the upper threshold T H is used in 
glasses frame image G 800. Images obtained through the above described 
procedures are I-component differential images £>'(/) 604 shown in Fig. 3. 

K,D(i)<T H 
then 

D , (0 = max(D(/),G(i)), 

where,/ = l,".,7V (Eq. 10) 

if, D(i)>T„ 

then 

D\i) = D(i) 

10 

I-component differential images D'(i) 604 are used for removing the 
occlusion regions due to the glasses I-component normalized images r(0 500. 
Error values in I-component differential images D\i) 604 have a gray-level range 
from 0 to 255. The error values are classified into non-occlusion regions, occlusion 
1 5 regions, and uncertain regions on the basis of the defined thresholds T L and T H . 

And then, according to Equation 11, a different weight is given to the non-occlusion, 
occlusion, and uncertain regions of I-component differential images D\i) 604. 



20 



\£,D\i)>T H 
then 
0*0 = 1 

if ,T L <D'(i)<T H 

then (Eq. 11) 

^) = l-0.5 r "- D ' ( '> 
T„-T L 

else 

a>(i) = 0 

wherein co(i) are weights with respect to I-component differential images D\i) 
604. 
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Weights of 1 are given to the occlusion regions having error values greater 
than the upper threshold T H and weights of 0 are given to the non-occlusion regions 
having error values smaller than the lower threshold T L . Herein, the weights of 0 
indicate that no change occurs in the original input facial images. Weights of 0.5 to 
5 1 are given to the uncertain regions. Although the lowest weights, i.e., 0.5, was 
determined by experiments, but not limit thereto, employing values capable of 
compensating unnatural looking facial images after removing the occlusions due to 
the glasses within I-component normalized images F(i) 500 is possible. This 
compensates only regions considered as parts of the glasses within I-component 
1 0 normalized images r(0 500. As described above, the weights (shown in Fig. 7) 
given by Equation 11 are used for compensating the occlusion regions due to the 
glasses within I-component differential images D'(i) 604 according to Equation 12. 

If (f = 0) 
then 

r/(0 = co cp + (1 - co) • r(0 (Eq. 12) 

else 

IT(0 = <» f f (o+(i-^)-r(o 

15 

wherein T t (J) is I-component compensated images 605 without bearing the glasses 

in accordance with the present invention. 

Where weights are 0, regions are determined as the non-occlusion regions 
so that I-component normalized images Y(i) 500 are used. Where weights are 1, 
2 0 regions are determined as the occlusion regions. In the case of the occlusion 

regions, at a first iteration (t=0) (i.e., a first column of Fig. 3) of the recursive PCA 
reconstruction in accordance with the present invention, I-component compensated 
images 605 are obtained using I-component average image cp 700, which is 
calculated from the sample facial images Y N . The reason why I-component 

25 average image cp 700 is used is that I-component reconstructed images f (i) 601 

are not glassless images in which the glasses included in I-component normalized 
images Y(i) 500 are completely removed. Accordingly, from the second iteration 
(i.e., the second column shown in Fig. 3), the glasses are removed using I-component 

reconstructed images f (/) 601, which are obtained by reconstructing I-component 
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compensated images 605 obtained at the first iteration (t=0). 

Where weights have a range from 0.5 to 1, regions are determined as the 
uncertain regions. Thus, the glasses are removed such that the weights are applied 
to the combination of I-component normalized images T(i) 500 and I-component 
5 average image cp 700, or I-component normalized images Y(i) 500 and I- 

component reconstructed images f (i) 601. Specifically, at the first iteration (t=0) 

of the inventive recursive PCA reconstruction, the glasses are removed by using the 
sum of resultant values upon multiplying a weight co to gray- level intensities of the 
uncertain region within I-component average image q> 700, and upon multiplying a 
1 0 weight (1- co ) to gray- level intensities of regions within I-component normalized 
image T(i) 500, the regions corresponding to the uncertain region within I- 
component average image cp 700. From the second iteration, the recursive PCA 
reconstruction is performed by using I-component normalized image 500 and 

I-component reconstructed image f (i) 601 reconstructed from I-component 

1 5 compensated image 605 at the first iteration (t=0). 

For example, in the first column of Fig. 3 except I-component normalized 

image T(0 500, I-component reconstructed and compensated images 601 to 

605 are generated such that finding the glasses regions within I-component 
normalized image r(i") 500 is performed and then processing the glasses regions is 
20 performed using I-component average image cp 700 as described above. I- 

component reconstructed and compensated images f (*) 601 to 605 in the second 

column of Fig. 3 are generated on the basis of I-component reconstructed image 

r(0 601 reconstructed from I-component compensated image 605 at the first 

iteration (t=0) and I-component normalized images 500. Such procedure is 

2 5 applied to subsequent columns from the third iteration. The inventive recursive 

PCA reconstruction is repeated until a difference between I-component reconstructed 

images f (i) 601 is less than a predetermined value 0 or it is constantly 

maintained as defined in Equation 13. 
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A A 



r,(o-r/ + ,(o \<e (Eq. 13) 



When the iteration of the recursive PCA reconstruction stops, compensated 
facial image, which is generated at the last iteration shown at right-bottom of Fig. 3, 
5 i.e., one of I-component compensated images 605, becomes an I-component final 
compensated image. Examples of I-component final compensated images are 
shown in Fig. 9. Hereinafter, the I-component final compensated image will be 
referred as an T image. 

Referring to Fig. 8, which is a flow diagram for processing S- and H- 

10 components normalized images obtained at step S216 in accordance with the 

inventive recursive PCA reconstruction. In order to obtain glassless color images, it 
is necessary to perform the recursive PCA reconstruction with respect to not only the 
I-component normalized images as described with reference to Fig. 3, but also the S- 
and H-components normalized images. Unlike the afore-mentioned recursive PCA 

1 5 reconstruction with respect to the I-component normalized images, glasses frame 

image G 800 extracted at step S212 shown in Fig. 2 is not used in the recursive PCA 
reconstruction with respect to the S- and H-components normalized images. The 
reason why glasses frame image G 800 is not used that occlusion regions obtained 
from the S- and H-components images are different from those of the I-component 

2 0 normalized images. 

First, detailed description of the recursive PCA reconstruction with respect 
to the S-component normalized image will be followed. Hereinafter, the S- 
component normalized image shown in Fig. 12 will be referred as an S-component 
normalized image F(i) . Before performing the recursive PCA reconstruction, 

2 5 image values of the S-component normalized image containing glasses are 

stretched to have values from 0 to 255. 

At step S802, image processing unit 1004 shown in Fig. 11 reconstructs the 
S-component normalized image , which is obtained at step S216 of Fig. 2, by 
using an S-component average image <p and eigenfaces u k through Equation 5, to 

3 0 thereby create an S-component glassless reconstructed image T(i) . The S- 

component average image <p and eigenfaces u k are calculated on the basis of the 
sample facial images Y N stored on face DB 2000 shown in Fig. 11. 

At step S804, image processing unit 1004 obtains an S-component 
differential image d(i) from the S-component normalized image T(i) and the S- 
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component reconstructed image f (/) generated at step S802, by using Equation 7. 

At step S806, the image processing unit 1004 stretches the S-component differential 
image d(i) by using Equation 8, to thereby obtain an S-component differential 
image D(i) in which facial features are reflected. 

As step S808, image processing unit 1004 calculates an average of errors 
within non-occlusion regions of the S-component differential image D(i) by using 
Equation 9, to thereby obtain the lower threshold T L . Image processing unit 1004 
further computes an average of errors greater than the lower threshold T L within the 
S-component differential image D(i) , to thereby obtain the upper threshold T H , at 
step S808. 

At step S810, image processing unit 1004 applies a weight of 1 on occlusion 
regions of the S-component differential image D(i) , a weight of 0 on the non- 
occlusion regions, and a weight having a value of 0.5 to 1 on uncertain regions by 
using Equation 10, to thereby generate an S-component compensated image. 
Similar to the processing of the I-component normalized image, an image 
reconstructed from the S-component compensated image, which is generated at the 

first iteration (t=0), is used as an S-component reconstructed image f (0 at the 

second iteration of the recursive PCA reconstruction. 

At step S812, image processing unit 1004 determines, using Equation 13, 
whether a difference between the S-component reconstructed images generated by 
the recursive PCA reconstruction is less than or equal to a predetermined value 0 . 
Where the difference is less than the predetermined value 0 , image processing unit 
1004 stops the recursive PCA reconstruction to obtain the current compensated 
image as an S-component final compensated image (hereinafter, referred as an S' 
image). The S' image is shown in Fig. 12. Otherwise, image processing unit 
1004 returns to step S802. 

An H-component normalized image is processed similar to the S-component 
normalized image described with reference to Fig. 8. Herein, H-component is 

represented as vectors of (H x , H y ) T using Equation 14. 




H x =cos(H) 
H y =s'm(H) 



(Eq. 14) 
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As is well known in the art, in HIS model, H-component is expressed in an 
angular coordinate system in which a zero and 360 degrees show the same color 
different from I- and S-components. In case of applying the recursive PCA 
reconstruction to the H-component normalized image, it is necessary to do not 
discriminate a color at zero degree from a color at 360 degrees. Therefore, the 

present invention utilizes the vectors of (H x ,H y ) T instead of the H-component. 
H x - and H y -vector-component normalized images are shown in Figs. 13 and 14, 

respectively. Similar to the S-component normalized image, the H x - and H y - 

vector-component normalized images are stretched to have values of 0 to 255, before 
applying the recursive PCA reconstruction thereto. H' x - and H' y -vector- 
component compensated images obtained by processing the H x - and H y -vector- 
component normalized images according to the recursive PCA reconstruction are 
shown in Figs. 13 and 14, respectively. 

In order to obtain an H-component final compensated image (hereinafter, 

referred as an //' image), restoring //^-and H' y -vector-component compensated 

images which were stretched is performed to apply the recursive PCA reconstruction 
thereto. Using Equation 15, the //' image is obtained with respect to the H- 
component normalized image. 

H » _ H'x 

X ^H'Z+H' Y 2 (Eq. 15). 

Herein, since the H' x - and H' y -vector-component compensated images do not 

satisfy the relationship of H' x +H' y 2 , H' x is normalized to (h' x 2 + H' y )fi , thereby 

obtaining H x . Then, H' is calculated within a range of 0 to 360 degrees on the 
basis of H x . 

Using Equation 16, the HSI-RGB transformation is performed on the//', 
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S', and /' images. The S' and /' images are subjected to serve the restoring 
process similar to the H' image. After the HSI-RGB transformation, /?'-, G'-, 
and B' -component images are obtained. Using Equation 16, a final color image is 
obtained on the basis of the R'~, G'-, and B' -component images. This process is 
widely known in the art so that detailed explanation will be omitted herein. 




If,0<//'<120° 
then 

r = |(1 + [(5'cos #')/(cos(60° - H'))]) 

g = l-{b + r) 

if,120°<//'<240° 

then 

H' = H'-120° 

g = |(1+ [(5'cos ')/(cos(60° - //'))]) 

fr = l-(r+s) 

if,240°<H'<360° 

then 

//' = #' -120° 

^ = + [(5'cos //')/(cos(60° - *0)D 
r = l-(g+fc) 

fl' = 3/V 
G' = 3/'g 
5' = 3/'& 



wherein r, g, and b are normalization values within a range of 0 to 1, which satisfy 
r+g+b=l, respectively. Equations 6 and 16 are well known in the art and detailed 
explanation will be omitted herein. (See R. C. Gonzalez and R. E. Woods, "Digital 
Image Processing," Addison- Wesley Publishing Company, 1992). 
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INDUSTRIAL APPLICABILITY 

As described above, in accordance with the present invention, natural 
looking high quality glassless color frontal facial images are obtained by 
simultaneously finding occlusion regions to be removed within input facial images 
5 and compensating the same. The occlusion regions include not only a glasses frame 
region but also reflection and shading regions due to lens and glasses. Further, the 
image processing method in accordance with the present invention is used to resolve 
other various occlusion problems and improves the recognition efficiency of 
automatic face recognition systems. 
1 0 While the present invention has been shown and described with respect to 

the particular embodiments, those skilled in the art will recognize that many changes 
and modifications may be made without departing from the scope of the invention as 
defined in the appended claims. 
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